Scheduling Hyperparameters to Improve Generalization: From Centralized SGD to Asynchronous SGD

This paper 1 studies how to schedule hyperparameters improve generalization of both centralized single-machine stochastic gradient descent (SGD) and distributed asynchronous SGD (ASGD). augmented with momentum variants (e.g., heavy ball (SHB) Nesterov’s accelerated (NAG)) has been the default optimizer for many tasks, in environments. However, advanced variants, despite empirical advantage over classical SHB/NAG, introduce extra tune. The error-prone tuning is main barrier AutoML. Centralized : We first focus on show efficiently a large class generalization. propose unified framework called multistage quasi-hyperbolic (Multistage QHM), which covers family as its special cases (e.g. vanilla SGD/SHB/NAG). Existing works mainly only scheduling learning rate α ’s decay, while QHM allows additional varying factor), demonstrates better than . convergence general nonconvex objectives. Distributed then extend our theory (ASGD), parameter server distributes data batches several worker machines updates parameters via aggregating batch gradients from workers. quantify asynchrony between different workers (i.e., staleness), model dynamics iterations differential equation (SDE), derive PAC-Bayesian bound ASGD. As byproduct, we moderately helps ASGD generalize better. Our strategies have rigorous justifications rather blind trial-and-error theoretically prove why could decrease derived errors cases. simplify process beat competitive optimizers test accuracy empirically. codes are publicly available https://github.com/jsycsjh/centralized-asynchronous-tuning.